To generate an effective visual dictionary for representing the scene of images, and further improve the accuracy of semantic annotation, a scene annotation model based on Formal Concept Analysis (FCA) was presented by means of an abstract from the training image set with the initial visual dictionary as a form context. The weight value of visual words was first marked with information entropy, and FCA structures were built for various types of scene. Then the arithmetic mean of each visual word's weight values was used to describe the contribution among different visual words in the intent to the semantic, and each type of visual vocabularies for the scene was extracted from the structure according to the visual vocabularies thresholds. Finally, the test image was assigned with the class label by using of the K-nearest method. The proposed approach is evaluated on the Fei-Fei Scene 13 natural scene data sets, and the experimental results show that in comparison with the methods of Fei-Fei and Bai, the proposed algorithm has better classification accuracy with β=0.05 and γ=15.